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Abstract 

We present a quantum algorithm for solving the hidden subgroup problem in the 
general linear group over a finite field where the hidden subgroup is promised to be a 
conjugate of the group of the invertible lower triangular matrices. The complexity of 
the algorithm is polynomial when size of the base field is not much smaller than the 
degree. 

1 Introduction 

The hidden subgroup problem (HSP for short) is the following. We are given a black box 
function / on a group Q such that there is a subgroup "H of ^ satisfying f{x) = f{y) if and 
only if X and y are in the same right coset of H (that is, yx"^ G H). The task is to determine 
the subgroup T-L, which is unique and called the subgroup hidden by /. Computing orders of 
elements of groups, calculating discrete logarithms and even finding isomorphisms between 
graphs can be cast in the paradigm of the HSP [18] . 

On classical computers, the query complexity of the hidden subgroup problem is expo- 
nential (in log 1^1) already over finite commutative groups. In the quantum setting / is 
assumed to be given by a quantum oracle which is a unitary map Uf mapping states of the 
form \x)\0) to \x)\f{x)). In contrast to the classical case the quantum query complexity of 
the HSP is polynomial (in log \ Q\), see [9]. Furthermore, there are polynomial time quantum 
algorithms [11 [19] solving the hidden subgroup problem in abelian groups, generalizing Shor's 
result on order finding and computing discrete logarithm [22j. Computing the structure of 
finite commutative black box groups [6J is a more general application of the abelian HSP. 
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As the graph isomorphism problem is involved, in the past decade considerable efforts 
have been spent on finding efficient algorithms for noncommutative cases of the HSP. Al- 
though nice results have been achieved in this direction, the groups in which the HSP can 
be solved at present in quantum polynomial time remain actually very close to being com- 
mutative. One of the widest classes of finite groups in which the HSP is known to have a 
polynomial time quantum solution consists of solvable groups whose derived subgroup have 
of constant derived length and constant exponent [lOj. Other classes of groups with efficient 
quantum HSP algorithms include certain "almost Hamiltonian" groups [TT] and two-step 
nilpotent groups [T7]. The latter class contains Heisenberg groups for which efficient HSP 
algorithms are also given in [2] and [1]. The "pretty good measurement" technique of [2] 
actually works also in certain nilpotent semidirect product groups of higher nilpotency class. 
An efficient HSP algorithm is given in [15] for a special family of groups which possess a 
large commutative subgroup and a map transforming the HSP of the whole group to the 
HSP of the abelian subgroup. The limited success in finding good noncommutative HSP 
algorithms indicates that the problem may be actually difficult. The connection between 
the HSP in dihedral groups and some supposedly difficult lattice problem exposed in [21j 
provides further evidence for that. 

Putting restrictions on the class of the possible hidden subgroups can result in efficient 
algorithms for finding them even in fairly noncommutative groups. Most importantly, the 
HSP for normal subgroups can be solved in quantum polynomial time in groups for which 
efficient quantum Fourier transforms exist (see [IS] and [H]) and in a class of groups including 
solvable groups and more [16]. The methods of [20] and [12] work efficiently for sufficiently 
large non-normal hidden subgroups in certain semidirect products. 

The first polynomial time algorithm for finding special hidden subgroups in simple and 
almost simple groups is given in [7]. (An almost simple group has a large noncommutative 
simple constituent.) The main result of Ibid, is an efficient quantum algorithm that solves 
the HSP in the group of 2 by 2 invertible matrices (and related groups) where the hidden 
subgroup is promised to be a so-called Borel subgroup (definition will be given below). In 
this paper generalize this result to finding hidden Borel subgroups in general linear groups 
of higher degree. 

We denote by GL„(Fq) the general linear group consisting of the invertible nxn matrices 
over the finite field Fq having q elements. We propose a quantum algorithm for the HSP in 
GL„(Fq) where the hidden subgroup is promised to be a Borel subgroup. For brevity we use 
the term hidden Borel subgroup problem for this promise problem. Our algorithm works in 
polynomial time if q is not much smaller than n. 

A Borel subgroup of GL„(Fg) is a conjugate of the subgroup consisting of the invertible 
lower triangular matrices (see [23]). An alternative definition for a Borel subgroup is being 
the stabilizer in GLn{Fq) of a flag F'^ > Ui > U2 > ■ ■ ■ > Un-i > (0) of subspaces of the 
space y = F" of the column vectors of length n. Indeed, for < /c < n let be the set of 
column vectors whose first k entries are zero. Then the invertible lower triangular matrices 
A form the stabilizer of the flag F^ > Vi > V2 > . . . > V^-i > (0), and their conjugates 
X~^AX by X form the stabilizer of the flag F^ > X'^Vi > X-^V2 > ...> X'^Vn-i > (0). 
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To fix a nicely defined output, by solving the hidden Borel subgroup problem we mean 
determining the fiag of subspaces stabilized by the hidden Borel subgroup. We remark that, 
given such a fiag, it is easy to construct generators for its stabilizer. 

Our method (described in Section [3]) is based on the observation that a coset of a Borel 
subgroup is quite a large subset of a linear space of n by n matrices. The main technical 
tool is a version of the standard algorithm for the abelian HSP, adapted to linear spaces (see 
Section [2]). In Section H] we show how to extend our result to finding hidden Borel subgroups 
of the special linear group. 



2 The quantum Fourier transform for linear spaces 

In this section we briefiy overview the main ingredient of the standard method for solving the 
hidden subgroup problem in F™ where the hidden subgroup is promised to be an Fg-linear 
subspace W of and give an interpretation of the result in the special case of a linear 
space of matrices. 

The procedure receives a superposition 

^ El^ + ^o) (1) 



over a coset W + vq and obtains information on W using the quantum Fourier transform 
(QFT) of the group F^. Here we use a version which is the m'th tensor power of the QFT 
defined in [8] for F^. This transform maps a |x) (x G F^) to 

2Tvi 

where Tr is the trace map from Fq to Fp and u is the primitive p'th root of unity e p . 
(Here p is the prime such that q = for a positive integer r and the trace map is defined 
as Tr(2;) = I][=o x^"^.) By Lemma 2.2 of [8], this map has a polynomial time approximate 
implementation on a quantum computer, therefore its m'th tensor power can be efficiently 
approximated as well. The image of \x) for a vector ) € F™' under the 



tensor power map is 



^m/2 



where (x, y) stands for the standard scalar product x^y = J^iLi ^iUi on F™. Our input 
superposition ([T]) gets mapped to the state 

E Cyb), 



where 



= f—, E 



iWlq'^v&w 
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The subspace W-^ consisting of the vectors u from such that {u, v) — for every v &W 
has dimension m — dimF^ W, therefore \ W-^\ — For y e W-^ we have 

\Cy\ — } 00 = = 

It follows that 

Therefore for y W-^ we have Cy — and if we measure \y), we obtain a uniformly random 
element of W-^. 

Assume now that W is a subspace of the linear space Alnxn(Fg) of n x n matrices over 
Fq. We can consider nxn matrices as vectors of length n^. Then the standard scalar product 
of two matrices A — (aij) and B — (bij) is 

n 

Here, for a matrix D G A4nxn{Fq), by tr(D) we denote the sum of the diagonal elements 
of D. (Thus tr(D) is an element of F^. The map tr from A^„xn(Fq) to Fq should not be 
confused with the trace map Tr from F^ to Fp, although they are not completely unrelated.) 
We will make use of the identity Tt{XY) = Tr(yX). 



3 Finding hidden Borel subgroups in the general hnear 
group 

In this section we outline a quantum algorithm for finding a hidden Borel subgroup H in 
the group GLn(Fq). Like the most hidden subgroup algorithms, our procedure is based on 
using superpositions over cosets of T-L, that is, states of the form 

where 5 is a matrix from GL„(Fg). We will think of such a superposition as an approximation 
of a superposition over a linear space of matrices and apply the quantum Fourier transform 
of the linear space A4nxn(Pq) to obtain a guess for the last subspace in the flag stabilized by 
H. The guess will be verified in a straightforward way. If the guess turns out to be correct, 
the further members of the flag can be obtained by recursion. 
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3.1 Obtaining coset superpositions 



The standard approaches to the hidden subgroup problem in a group G start with the the 
state — ^ J2xGg ^-PPly the oracle for the function / to obtain — ^ J2xeg \^)\f{^))^ 



finally measure the second register to obtain the coset superposition 



rL\ x&H 



with some y & Q {Ti is the subgroup hidden by the function /). If ^ is abelian then the 
uniform superposition — ^ J2x&g \^) over Q can be obtained by applying the quantum Fourier 



transform of ^ to |0) (here stands for the neutral element of ^). There are efficient methods 
for computing uniform superpositions of certain further classes of groups, e.g., the algorithm 
of Watrous ^24j for solvable groups. 

For the purposes of our algorithm it will be sufficient to approximate the uniform su- 
perposition 
1 



■y/|A4„Xn(F,)| 



^X&M„xn(Fg) 



J2x£GLn(Fq) \x) over the group GL„(Fg) by the uniform superposition 
x) over A^nxn(Fg), which can be efficiently computed using the 



quantum Fourier transform of . The fidelity between the two states is 



|GL„(F,) 

^ \Mnxn{F, 



\ 



n 



n(i 

i=i 



\ 



> 



Therefore, if we first apply the quantum oracle Uf to the superposition 
, =y]rpA/f (-F ) \x)\0) and then measure the second register, we obtain a super- 

position over a coset of the hidden subgroup "H in GL„(F5) with probability at least |. 



3.2 Guessing the last subspace in the flag 

Recall that our assumption is that there exists an n x n invertible matrix X such that 
n = X-^CX, where 

C = {A= (ttij) e GL„(Fg) : ttij = when i < j} . 

Then the last nontrivial member Un-i of the flag F]^ > Ui > . . . > Un-i > (0) stabilized by 
"H is X~'^Vn-i, where Vn~i consists of the column vectors from F^ whose flrst n — 1 entries 
are zero. 

We will consider the multiplicative group C as an approximation of the subspace 

C = {A= {ttij) e MnxniFq) : ttij = wheu i < j} 

of lower triangular matrices. Then "H will be thought of as an approximation of TL' = X~^CX. 
We have \n\ = \C\ = (q - l)"g«("-i)/2 and \n'\ = \C'\ = g"("+i)/2. 
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Accordingly, for every B e GL„(F^), the coset superposition 



will be considered as an approximation of 



in'B) = J2 \X-^AXB). 



The fidelity between \%B) and \H' B) is 



V,B\ _ ^\V,\ _ (q-iy 

^\n'B\ ,J\w\ \ <i J ' 

Therefore, if we apply the quantum Fourier transform of Mnxni^q) discussed in the 
previous section to the coset superposition iTiB), and do the measurement then, with a 
chance at least ((^7^) )' the result will be a uniformly random element of the subspace 
(H'B)^, as it would be the case when we started with the state \'H'B). 

Let F be a matrix from Mnxn{^q)- Then Y e {WB)^ = (X-^C'XB)^ if and only if 
tr{X-'^AXBY^) = for every A e C. As 

tr (^X'^AXBY^) = tr (^AXBY^X'^') = tr (^A ((X^)-^rS^X'^) 

we obtain that Y G {H'B)^ if and only if {X'^)~^Y B'^ X'^ G Furthermore, as multiplying 
matrices by B^ and conjugating matrices by arc bijections, we can conclude that if Y is 
a uniformly random element of {%' B)-^ then (X^j^^FSX^ is a uniformly random element 
of Observe that the elements of are just the strictly upper triangular nxn matrices. 
A strictly upper triangular matrix Z has rank n — 1 if and only if all the entries of Z just 
above the principal diagonal are nonzero. For a uniformly random strictly upper triangular 

Observe that if Z is a strictly upper triangular matrix of rank n — 1 then the kernel of 
Z'^ is the set Vn-i of column vectors from F** whose first n — 1 entries are zero. Obviously, 
the matrix {X^)~^Y B^ X^ has the same rank as Y. If the rank is n — 1, then the kernel 

of XBY^X~^ = (^{X^y^YBX^Y is ^n-u whence the kernel of Y^ is the 1-dimensional 
subspace X~^Vn-i, which is the last subspace of the flag stabilized by H. 

In summary, by applying the quantum Fourier transform to the coset state {HB) and 
then measuring the result, with probability O ((1 — we obtain a matrix Y of rank 

n — 1 with kernel Un-i- 
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3.3 Putting things together 

In this part we show how to check and use a guess for the last subspace Un-i of the flag sta- 
bilized by the hidden Borel subgroup H provided by the algorithm described in the previous 
subsection. 

As Un-i is one-dimensional, we assume that the guess is given by a nonzero column 
vector u. Let Z be a matrix from GL„(Fq) whose last column is u. Then = ZVn-\- 
We replace the hiding function / by /' defined as /'(^) = f{ZAZ~^). An oracle for /' can 
be obtained from the oracle for / in an obvious way using this definition. The subgroup 
hidden by /' is Z'^'H.Z and the last subspace of the flag stabilized by Z~^'KZ is Z'^Un-x- 
The guess for f/„_i is correct if and only if Z~^Un-\ = Vn^i, that is, the subgroup Z~^'HZ 
hidden by /' is contained in the subgroup of matrices of the form 

\ V a ) 

where A e GL„_i(Fq), a e \ {0} and v is row vector of length n — 1. Testing correctness 
of the guess can be carried out by calling the oracle for the identity matrix and for the n—1 
matrices of the form 

(;:) 

with V = (1, 0, 0, . . . , 0), (0, 1, 0, . . . , 0), . . ., (0, 0, . . . , 0, 1). Note that we can obtain a cor- 
rect guess with expected O {{1 — q~^)~"') repetitions of procedure described in the previous 
subsection. 

Assume that the guess is correct. Then we consider the subgroup Q of GL„(F) consisting 
of the matrices of the form 

with A e GL„_i(Fq). Taking the upper left n — 1 by n — 1 block of matrices gives an 
isomorphism between Q and GL„_i(Fg). Furthermore, Q fl Z~^HZ is a Borel subgroup of 
Q. The subspaces in the flag stabilized by ^ fl Z'^'H.Z are intersections of those for Z'^'HZ 
with the subspace of the column vectors with zero as last entry. We determine this flag by 
recursion. Then we obtain the flag for Z~^l-iZ by adding Un-\ to each of the members. 
Finally the complete flag for "H is obtained by applying Z. 

The group PGL„(Fg) is the factor of GL„(Fg) by its center consisting of the scalar 
matrices and the Borel subgroups of PGL„(Fg) are just the images of the Borel subgroups 
of GL„(F^) under the quotient map. As the scalar matrices from GL„(Fg) are contained 
in every Borel subgroup, the hidden Borel subgroup problem for the groups PGL„(Fg) and 
GL„(Fq) essentially coincide. (A function hiding a Borel subgroup of PGL„(Fq) can be lifted 
to GL„(Fq) in the straightforward way.) We have proved the following. 

Theorem 1 The hidden Borel subgroup problem in GL„(Fg) (and in PGL„(Fg)^ can be 
solved in quantum time poly(n -|- logg + (1 — q~^)~"')- 
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4 Finding Borel subgroups in the special linear group 



The special linear group SL„(Fg) consists of n by n matrices over with determinant 
one. In this section we briefly outline an extension of our method to finding hidden Borel 
subgroups in SL„(Fq). A Borel subgroup of SL„(Fg) is just the intersection of SL„(Fg) with 
a Borel subgroup of GL„(Fg), that is, the stabilizer of a flag F^ > f/i > . . . > Un~i > (0) 
of subspaces within SL„(Fg). Again, we require the output of the HSP algorithm to be this 
flag. 

Assume that we have a function / defined on SL„(Fg) that hides a conjugate of the 
subgroup Co consisting of the lower triangular matrices having determinant 1. Let Q stand 
for the subgroup of GL„(Fq) consisting of matrices whose determinant are from F*", where 
F*" = {x" : ^ X e Fq}. We extend / to ^ as follows. Let A be a matrix from Q. We 
compute det A and find en element z E Fg such that = det A. Such elements z can be 
found e.g., by Berlekamp's polynomial factoring algorithm [31 H]. We put f{A) = f{z~^A). 
It turns out that the definition of f{A) does not depend on the choice of z. Indeed, if z^ = z^ 
then Zy^zI is in the subgroup of SL„(Fg) hidden by / and therefore /(^f ^A) = f{zA). The 
subgroup of Q hidden by the extended function will be a conjugate of the lower triangular 
matrices with determinant from F**^. The fidelity between the uniform superposition over 
this set and the uniform superposition over all the lower triangular matrices is at least 

(^^) ■ Therefore, if we apply the method of Subsection 13.21 for guessing the last element 

of the stabilized flag, we have a further factor for the probability of obtaining a correct 

guess. Testing correctness and the recursion are also essentially the same as in the case for 
GL„(Fg). We obtain the following. 

Theorem 2 The hidden Borel subgroup problem in SL„(Fg) (and in PSL„(Fg) ) can be solved 
in quantum time poly(?2 + logq + (1 — q^^)^'^)- 

As (1 — q^^)^^ is polynomial in n if g = ^(j^^), we have 

Corollary 3 When q = ^(j^); the hidden Borel subgroup problem in GL„(Fq) and SL„(Fg) 
(and in PGL„(Fq) and PSL„(Fg)j can be solved in quantum time poly(n + log g). In partic- 
ular, for constant n, the quantum complexity of the problem is poly(logg). 

5 Concluding remarks 

In this paper we have proved that the hidden Borel subgroup in GL„(Fg) and SL„(Fq) can 
be solved in quantum polynomial time if the size q of the base field is not too much smaller 
than the degree n. Perhaps the most important question which if left open is existence of 
polynomial time algorithms over small base fields (e.g., over fields of constant size). 

Other interesting questions are whether it is possible to extend the result to the hidden 
Borel subgroup problem in other classical groups (e.g., the orthogonal groups) and if it is 
possible to find efficiently hidden conjugates of certain subgroups of the lower triangular 
matrices such as the unitriangular matrices or the diagonal matrices. 
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